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One way to enhance the effectiveness of Special 
Purpose Operational Training 1 (SPOT) debriefing 
sessions may be for instructors to make explicit 
connections between the Crew Resource Management 
(CRM) concepts a carrier advocates and the behaviors 
displayed by the crew in question. A tool listing key 
behaviors from the scenario was devised, accompanied 
by an instructors’ training session in which links were 
made between the behaviors and the underlying CRM 
processes they reflect. The aim of the tool is to assist 
instructors to focus the debriefing on the key SPOT/ 
CRM issues, in this case on planning. A second tool 
suggested ways to facilitate the discussion. 

Fourteen instructors at a major U.S. earner took part in 
the training session and used the toolkit in their 
subsequent debriefs. Pre- and post- training debriefing 
samples from each instructor were compared to assess 
whether there were any changes in instructors 
approaches to discussions in terms of the topics they 
covered and how they raised the points. 

CRM DEBRIEFING 

CRM has prompted a ‘growth spurt’ in flight training 
over the last 20 years. Like most new concepts, 
developers and airlines have progressed through cycles 
of testing and reworking to refine their CRM training 
into an effective package whose effectiveness will be 
borne out through an increase in safe behaviors in 
flight. Helmreich (1996) suggests that CRM has now 
entered its fourth phase of development. In practical 
terms, this means that most U.S. airlines have an initial 
awareness/training package in place in their fleets. 
Also, their crews have attended a number of recurrent 


1 SPOT is one of three types of Line Operational Simulation (LOS) 
defined by the FAA (1990). SPOT, like LOFT (Line Oriented 
Flight Training), focuses on crew training. 


training sessions. Hence, all the key CRM skills 
identified by the airline have been ‘refreshed’ (Flin & 
Martin, 1998). Airlines are now looking for evidence 
that their training is transferring to line operations. 
While awareness of CRM is high, some would argue 
(Komich, 1997) that these skills are not transferring to 
the line. A theme of this fourth phase of development 
has become to increase the effectiveness of CRM 
recurrent training, that is, to encourage this transfer of 
skills. 

Debriefing of crew performance in Line Oriented 
Flight Training (LOFT) has been identified as a key 
opportunity for promoting the transfer of CRM skill 
(Helmreich & Foushee, 1993). Through their active 
participation and discussion of CRM concepts in the 
simulator scenario they just flew, crews leam how they 
themselves perform CRM and how they might enhance 
their techniques. The content of recurrent training is 
predominantly in operationalizing concepts rather than 
learning new ones (Smith, 1994). However, Dismukes, 
Jobe and McDonnell (1997) found that crew 
participation, and hence the transfer of training, may 
not be as ‘simple’ as exposing crews to CRM concepts 
and a variety of relevant practice sessions. Dismukes 
et al. identified the process of debriefing rather than the 
session content as a central problem at U.S. carriers. 
Instructors and crews may be unsure how to discuss 
CRM concepts embedded in scenario events in a way 
that helps the crew to operationalize and apply them to 
their own behavior. 

The difficulties found with CRM debriefing suggest 
that both instructors and crews may benefit from tools 
that explicitly make the connection between CRM 
concepts and behaviors in a recurrent training program. 
Each event in the scenario involves a multitude of 
behaviors. Some of these will serve purely technical 
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recment°may make it difficult for all those involved to 
sift out the most appropriate behavioral exMip es or 
discussion (Tourville, 1997). Therefore a list of key 
behaviors may assist debriefing participants in their 

discussion of CRM. 

Another route to achieve operationalization of CRM 
concepts is to draw on the strengths ofadultle^ g- 
Adults engage m participative leam g, ^ 

‘do’ or try out, what they are learning (Cornwell, 
1979) LOFT capitalizes on this by focusing on 
Italud flight (Smith, 1994). This 
should extend into the debriefing session, ldeally ”1* 

,he erew dricussing their flight end tht 
as a facilitator (Lauber & F oushee, 1981). 
al (1997) found that instructors and crews were n 

confident or clem about 

facilitation To this end, McDonnell, Jobe 
Dismukes (1997) produced guidelines for fam tetion 
that pragmatically address some of the issues arising 

£L P S reseLh. Once again, instructors muy 

benefit from specific examples of facl '“ 
thev can use for the scenario they are debriefing. 
KSng how die instructor facilitates, crews may 
develop a clearer understanding of the issues they 
when they are self-debriefing (Butler, 

1993). 

From the review by McDonnell et al. (1997), it appears 
that CRM is an area where crews and instructors no 
only have to deal with concepts that are stlll /^ lat ' ve ^ 
new to them and which they are try m g 
operationalize, but they are also debriefing the CRM 
training events using unfamiliar metho . 

SPOT debrief is a double learning event for crews. 

Below we describe preliminary findings from a study 
mtpkuing methods and tools for « 
operationalization of CRM concepts in L ^ T ^ree to 
Tlhe analysis reported here focused on the degree to 
Sf too* deseed to meet^he needs 

identified above had an impact o 
debriefings for one fleet of a major U.S. earner. 

METHOD 

A SPOT toolkit was developed during the initial phase 
* to study (see Villeda, Martin & Orasanu, m press 
for details). Two of the tools in the kit were designed 
to support LOFT instructors: an observation-debne mg 
guSe which noted event items with CRM significance, 
Ld a strategies list which provided questions and tips 


to encourage a facilitated style of discussion^ A third 
tool was directed toward the pilots: a self - debn ® ff °™ 
containing the same items as the ° bse ^^^ ef an ® 
guide developed for the instructors. The toolkit and 
how it could be used to support debriefings was 
presented to instructors in a group training session, 
ore-post within-subjects design was used. Baseline 
SPOT debriefings were observed before the tootat was 
introduced, essentially providing a cart'd conditmm 
The study condition consisted of SPOT debriefings 
after the toolkit was introduced for the instructors use. 

The study population included instructors and crews 
from 1 fle* * the participating airline. A random 
sample of 92 recurrent debriefings was collected, 
Sving 33 instructors and 184 pilots. Fifty-one were 
baseline or control sessions and 41 were post-tramuig 
oHtudy sessions. For the purpose of a prelimm-y 
analysis a sub-sample of 14 instructors was selected. 
Oi^eime and one post-training deb„=fmg «ss.on 
from each of the selected instructors served as the dam 
source for this initial analysis (28 sessions), 
held tape recorders were used to obtain audiotap 
recordings of the SPOT debriefs. 

The study tapped into current SPOT training practices 
m pSicipalrng airiine. Tims, the — « 
package was not changed for participants in th y 
Recurrent training at the participating airline consists 
“Iday program. A ground-school session on 
Ae first day mcludes review of the CRM concepts to 
£ practiced in the SPOT. The SPOT flight m-d 
debriefing occur on the second day, the focus of our 
S Aey are separated by a two-hour maneuvers 
validation session. The third day involves proficiency 
check rides (Line Operational Evaluation). 

RESULTS 

The 28 audiotapes were partially transcribed and, for 
this preliminary analysis, coded according 
content, or topics, of discussion and who raised them. 


Overall Content of Debriefs 


Three hundred mid twenty nine topici ‘ 
during the debriefings under study (X 11-75 p 
debrief). These topics cluster into 73 themes, 
average 1.7 more topics were discussed in the stu y 
condition than during to baseline debrmfr 
(Xbaseline=10.9, Xtoolkit=12.6, see Table 1). 
distribution of the discussion in terms of conte ^ 
also different between the two condl “°" ' ± - 

baseline condition, participants spent 36.1/. of then 
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discussion talking about technical issues; after the 
training session this proportion reduced to 30.5%. As a 
balance, the proportion of CRM talk increased after the 
training session from 55.3% to 62.7%. Although these 
differences are not significant, due perhaps to the small 
preliminary sample size, they are in the desired 
direction. CRM discussion increased to replace the 
decrease in technical discussion. Particularly 
interesting are the results from the planning 
subcategory, as this was the focus of the SPOT being 
observed. Table 1 illustrates that half of the increase in 
CRM discussion is accounted for by an increase in 
planning talk. 

Mention of Specific Observation Tool Items 

References to the behaviors identified on the 
observation tool were collected from the partial 
transcripts. Comparisons were made between the 
baseline and post-training conditions. On average, 5.9 
behaviors from the observation tool were discussed in 
the baseline debriefs; 8.7 behaviors were raised in the 
toolkit condition on average, an increase of nearly 3 
behaviors per debrief (see Figure 1 for the number of 
items discussed in each of the 28 debriefs). In 
particular, the proactive and reactive p lannin g 
behaviors were compared (planning was the training 
focus this year). Mentions of proactive planning items 
increased from an average of 1.7 in the baseline to 3.2 
in the toolkit condition. Mentions of reactive planning 
items increased from an average of 0.8 in the baseline 
to 1.9 in the toolkit condition. Thus, for both types of 
planning items, the discussions approximately doubled 
after the toolkit was introduced). 

Topic Initiation 

Additional analyses revealed who was talking during 
the debriefings (see Table 2). Of 152 topics raised 
across the 14 baseline debriefings, 147 were initiated 
by the instructor and 5 by the participating crew. In 
the toolkit condition, the instructor initiated 1 60 of the 
177 topics, with 17 initiated by the participating crew. 
This constitutes a three-fold increase in the discussions 
initiated by the crews after introduction of the toolkit. 
These frequencies were compared through a Chi- 
Square test, which gave an AC 2 value significant at the 
p<025 level ( X 2 =5 .227, d/'=l). Another post-training 
increase was the number of discussions in which the 
instructor prompted the crew to talk about an event 
within the scenario that they thought was significant on 
a CRM-related parameter (e.g. difficult or stressful). 
The number of discussions in which the crew selected 


the topic increased from two in the baseline condition 
to 13 in the post-training condition. 

TOOL ENHANCEMENT OF DEBRIEFINGS 

Each of the analyses reported above directly addresses 
one of the needs identified earlier in this paper. One 
aim was to increase the amount of CRM talk in SPOT 
debriefings, so participants would be making explicit 
CRM-behavior connections. 

Structuring the content of the 28 debriefs showed that 
the number of topics discussed increased only slightly, 
but that the content of the discussions had shifted. The 
proportion of topics devoted to CRM increased on 
average. Planning, the focus of these SPOT sessions, 
accounted for half of this increase. Technical talk 
showed a corresponding reduction in the proportion of 
the debrief it occupied. 

These results suggest that the toolkit helped instructors 
to focus their debriefings on the CRM aspects of the 
SPOT, thereby discussing the impact of the crews’ 
CRM behaviors (e.g., planning, workload, 
communication, etc.) on the events. The training 
benefit of this increase in CRM-talk is that more of the 
behaviors crews performed in the simulator will be 
actively linked to CRM concepts. More instances of 
crews’ real-world experiences will be coupled with 
terms and ideas of resource management as examples 
of how they practice CRM. 

It is not possible to specify which items in the toolkit 
were most influential in shifting the focus and structure 
of the debriefing discussions. It is possible that the 
training session reminded the instructors of the purpose 
of the training; that the observation tool prompted 
instructors to talk more about behaviors with specific 
CRM content; or that instructors used more effective 
strategies for eliciting crew participation. It is also 
possible that the observation tool acted as a memory 
aid for the instructors, reminding them of behaviors 
they saw during the SPOT that had noteworthy CRM 
content. Most likely, these influences worked in 
combination. 

One of the methods which may have promoted the 
increased CRM discussion was the use of the 
observation tool by the instructors. Mention of the 
behavioral items, each of which was selected for the 
tool because of its important CRM content, increased 
by an average of 2.8 items per debrief. The 11 
planning items accounted for the majority of this 
increase (81%). The introduction of the toolkit 
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therefore led to a greater amount of planning 
discussion during the debriefs. In particular, this 
planning discussion reinforced the link between 
planning concepts and planning actions through 
mention of the key behaviors from the observation 
tool. 

For example, reactive planning, necessary when 
unexpected events occur, received little discussion time 
in the baseline debriefs. In nearly half (6), there was no 
mention of reactive planning in any form. The modal 
number of reactive planning items increased to two 
post-training. Only two of the post-training debriefings 
contained no mention of any of the reactive planning 
items from the observation tool. It is possible that 
instructors found reactive planning a particularly hard 
concept to address and that the observation tool 
provided a way for them to raise reactive planning 
behaviors. 

Another aim of the toolkit was to encourage greater 
crew participation. Through conducting mini self- 
debriefs with instructor guidance, crews can leant how 
to debrief an entire facilitated session. One of the 
measures of greater participation is the number of 
discussion topics that are initiated by the crew. The 
crews in the baseline condition (Table 2) initiated 
around 3% of the topics raised, while crews in the post- 
training condition initiated three times as many 
conversations (just under 10% of the topics). This 
increase approximates the proportions found when 
topics that the instructor prompted with an open-ended 
question are included (e.g., “Tell me about an event 
you found difficult”). In these latter cases, the crew 
chose both the event and the focus of the discussion 
(e.g., why they found it difficult). Increases in the 
number of crew-selected topics reflect the instructors’ 
use of facilitation skills to encourage the crew to 
debrief themselves. These kinds of prompts are 
included in the debrief strategy tool that was part of the 
toolkit. Therefore, these instances reflect instructor use 
of the toolkit as well as indicate an increase in crew 
participation. 

The greater initiation of topics by crews in the study 
condition may also reflect the influence of the crew 
self-debrief tool. The self-debrief tool may have 
helped crews follow the instructor’s line of questioning 
and may have helped them recall the events from the 
SPOT. Dismukes et al. (in press) note that crew 
members in their interviews mentioned difficulty 
remembering a SPOT scenario which has occurred a 
number of hours earlier and is followed by other 
simulator training exercises. The carrier participating 


in the present study runs all the recurrent simulator 
training at one time and debriefs all the sessions 
afterwards, so the crews we observed may have 
experienced similar memory problems. 

CONCLUSIONS 

These preliminary findings suggest the toolkit 
improved SPOT debriefings on a number of levels, 
providing assistance to both the crews and to the 
instructors. They suggest that the toolkit helped 
instructors to refocus the topics they raised onto 
behaviors with a CRM content and in particular to 
provide reminders about key planning behaviors. 
These connections may help crews to operationalize 
the CRM concepts. The toolkit may also have had an 
effect by providing the instructors with some simple 
strategies (forms of questioning) they could use to 
prompt crews to begin to take the lead in debriefing 
themselves. By using these strategies the instructors 
were also beginning to train the crews in self- 
debriefing methods. 

The findings suggest that the toolkit also provided a 
memory aid for the crew, reminding them of issues that 
they wanted to raise and increasing the learning value 
of the SPOT. Having remembered their actions in an 
event, a discussion of relevant CRM issues provides 
crews with a closure of the learning cycle. Crews, by 
discussing their actions, are engaging in participative 
learning. Their discussion links the theoretical concepts 
covered in ground-school to actions that they 
performed, providing the crew with examples of the 
way they communicate or manage their workload. 

Confirmation of these findings awaits a complete 
analysis of the full data set, which is currently 
underway. This analysis will provide more detail on 
how the debriefing tools influence the structure and 
content of SPOT debriefs. 

The results regarding the content, the initiators of 
debrief topics, and the references to key behaviors 
support the aims of the toolkit. These were to increase 
the amount of CRM talk during the debrief sessions, 
particularly about planning, and to increase the amount 
of crew participation in the session. All the aspects of 
the debriefs we studied increased after the toolkit was 
introduced to the instructors at the participating airline. 
The training benefit of these changes to the debriefings 
may be to increase the degree to which crews reflect on 
their own actions in CRM terms and to prepare them 
for self-debriefing. The ultimate goal is for self- 
reflection and analysis to carry over to the line. 
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